:: JKMS :: Journal of Korean Medical Science

Over the years, patient-reported outcomes (PROs) have become important endpoints in clinical trials, health care, and the social sciences.1, 2 PROs are health outcomes that are directly reported by the patient without interpretation by a clinician or anyone else. Many studies have confirmed that PROs should be considered as the standard for measuring perceived health condition, symptoms, functional status, and health-related quality of life to adequately guide treatments and improve health outcomes.3, 4, 5, 6 In addition, PROs may provide information relevant to treatment choices, disease management, reimbursement decisions, and health policies in various clinical settings.7, 8

However, most PRO measurement instruments have encompassed problems such as a narrow scope of health status and difficulties in standardized interpretation owing to irrelevant, incomprehensible, or poorly formulated questions.9 Thus, in 2004, the Patient-Reported Outcomes Measurement Information System (PROMIS®) was established as a multi-center, collaborative project funded under the US National Institutes of Health Initiative to improve the measurement of clinically important symptoms and outcomes.10 The PROMIS is based on the conceptual framework advocated by the World Health Organization, and has three key domains: physical, mental, and social health.11 These domains have a set of item banks to facilitate standardized assessment of PROs in adults and children. Initial development of the constructs and domains of PROMIS is detailed elsewhere.12 The item banks for the measurement of a large number of clinically important outcomes can be administered either as short forms (fixed subsets of the best items from an item bank) or through computerized adaptive testing (where a computer algorithm selects items from an item bank one by one based on answers to previous items, leading to a variable subset of items per person). The item banks were developed using item response theory (IRT) methods, which model the relationship between the level of the domain and the probability of giving a certain response to an item. Based on the IRT model, items are ordered in an item bank based on the item difficulty and this ordering allows for the selection of subsets of items for short forms or computer adaptive tests (CATs) and ensures that all short forms and CATs are measured on the same scale and that scores can be compared directly.

The PROMIS has been translated and validated in various languages, and the domains of the adult item banks have been internationally adopted in clinical research and practice.13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23 The PROMIS has the potential to be global health scale considering as different item parameters could be used across countries and because its psychometric properties enable efficient assessments, which is valuable in clinical research and the evaluation of medical care.24, 25, 26 Thus, we aim to translate and linguistically validate a Korean version of the PROMIS (K-PROMIS) for six profile adult domains.

Translation procedure

We translated and validated six of the eight profile domains of the PROMIS (“fatigue,” “pain intensity,” “pain interference,” “physical function,” “sleep disturbance,” and “ability to participate in social roles and activities”; Supplementary Table 1) because the items in the depression and anxiety domains have already been linguistically validated. In addition, items available in Korean from Short Forms before this study (e.g., Short Form 4a items) were also excluded from the study.

Fig. 1 provides an overview of the translation and linguistic validation procedure. A total of 268 items from six profile domains were translated into Korean according to the Functional Assessment of Chronic Illness Therapy (FACIT) multilingual translation methodology including forward and back translations, independent reviews of translation quality, and pilot testing including cognitive interviews.27, 28 This methodology is consistent with the guidelines for translation of PRO instruments and previously published translation and validation procedures of the PROMIS for non-English speaking populations. During the translation, we focused on semantic and conceptual equivalence of meaning with the original English PROMIS.

First, each English source item was forward translated into Korean by three independent bilingual professional translators who have licenses for Korean-English interpretation and translation. This step focused on capturing the essence of items rather than performing literal translations. Then, a fourth independent bilingual translator who was not involved with the forward translation process selected one of the three translations or created a hybrid version of the translated items. The reconciled version was then back translated by another set of three bilingual professional translators fluent in both English and Korean. Translators did not have medical background. While they were provided brief background information about PROMIS measurement system before the translations, no other information was provided to the translators. For quality assurance of the translation procedure, we organized an expert group consisting of 10 health professionals (six clinicians, two nurses, and two behavioral scientists) who were native speakers of Korean and fluent in English. The expert group reviewed the forward translations, reconciliations, and back translations based on the concept and definition provided by the PROMIS Statistical Center and identified inadequate expressions and concepts in the translation as well as any discrepancies between the original source items in English and the translated Korean items. If the previous translation steps were unacceptable, the experts selected the most appropriate translation for each item or provided alternate translations. In addition, the Korean Language and Culture Institute reviewed the translation for spelling errors and conceptual equivalence. The harmonization and quality assurance of the pre-final translation were performed by the PROMIS Statistical Center to address consistency and conceptual equivalence with existing translations in other languages if applicable, as well as between items.

Cognitive interviewing procedure

We conducted cognitive debriefing interviews to determine whether participants found any of the K-PROMIS items ambiguous or difficult to understand. This is an interview method designed to evaluate the comprehensibility, ease of response, and acceptability of the terminology, phrasing, response options, and format of a newly developed PRO measure.29

Questionnaire scripts

Debriefing scripts were created to guide the cognitive interviews. Each interview was conducted using a standardized script consisting of open-ended questions developed by the researchers. Each script contained 27 to 35 items to avoid burdening the participants in a single interview. The interviewer asked a series of open-ended questions, following a script, seeking comments with regard to the item stem, the response options, and the time frame (e.g., in the past seven days, now, etc.). Prior PROMIS cognitive debriefing studies have successfully used a similar approach.18, 23, 30, 31, 32 Data concerning participants' sociodemographic characteristics, including marital status, educational level, and monthly household income, were also gathered by self-report. Clinical information was obtained from electronic medical records. We calculated the total time (in minutes) taken to complete the K-PROMIS survey, excluding any breaks taken during the interview.

Participants

Sixty-seven adults aged 18 years or older who had seen a physician for chronic diseases or pain within the past five years, who had no concurrent psychiatric condition or cognitive or other impairments (e.g., visual) that would interfere with completing an interview, and who could speak, read, and write Korean as their primary language were recruited. Prespecified enrollment goals included the following: at least 30% of participants with an educational level of high school or older than 65 years, approximately equal gender representation, and at least five participants per item.28 Accrual of the sample was monitored prospectively to achieve these enrollment goals.

Interviews

Two rounds of cognitive interviews were conducted. Sixty-one participants were included in Round 1. After making the necessary revisions based on feedback from Round 1, six participants were included in Round 2. The cognitive interviews were performed in a private area of the outpatient clinic and consisted of the following two parts: administration of the K-PROMIS survey composed of a subset of K-PROMIS items, and a semi-scripted debriefing interview with cognitive probing for comprehension, clarity, and ease of judgment. Interviews were conducted by two experienced interviewers in conducting cognitive interviews.

After completing the paper-and-pencil K-PROMIS survey, participants were asked to indicate items they found difficult to comprehend and those for which they had difficulty selecting a response option. The interviewer did not provide any assistance or advice and encouraged patients to complete the survey to the best of their ability on the basis of the instructions provided. After completing the survey, the cognitive debriefing interview was conducted. The script was similar to those used in previous PROMIS studies.30, 31, 33 Participants identified confusing text from the K-PROMIS items and rephrased them in their own words. The interviewer asked the participants to suggest alternative wording or phrasing to improve the clarity of items that were difficult to comprehend. The interviewer took detailed notes, and the interviews were recorded and transcribed for analysis.

Iterative cycles of analysis and retesting

For analysis of the individual K-PROMIS items, interview field notes and transcripts were compiled, abstracted, and summarized item by item. The interview data were examined for semantic and conceptual equivalence to the original English items. The expert group reviewed the results of each round of interview data analysis. The proportion of respondents exhibiting any level of difficulty or hesitation for an item or response option was calculated. K-PROMIS items that met the a priori threshold of ≥ 20% of respondents with comprehension difficulties in Round 1 were considered for rephrasing and retesting in Round 2.

Items that at least 20% of the respondents found difficult to comprehend were classified under either linguistic or cultural difficulties. Linguistic difficulties were categorized into four issues: 1) “Do not know the meaning of the word” referred to a situation where the participants did not know or understand the meaning of a few words or sentences; 2) “Ambiguous meaning” referred to a situation where the participants knew the words but had a different interpretation than the intended meaning; 3) “Confused context” referred to a situation where the participants had an idea of the words but were confused about the exact meanings or were uncertain about the meanings in the correct context; and 4) “Others” referred to difficulties not included in the above three categories. Cultural difficulties were categorized into three issues: 1) “Differences in concepts or unfamiliar activities” referred to a situation where the participants did not understand the phrasing owing to conceptual differences or unfamiliar experiences; 2) “Differences in lifestyle and living environment” referred to a situation where the participants had difficulties responding because of differences in lifestyles and living conditions; and 3) “Different metric units” referred to a situation where the participants had difficulties understanding metric units.

Item revision was considered on the basis of a detailed review of participants' responses and in the context of an effort to produce a final version that would be well comprehended by diverse respondents, including those who were older and had lower educational levels. Following this, the PROMIS Statistical Center performed the final quality review.

Ethics statement

This study was approved by the Institutional Review Board (IRB) of the Samsung Medical Center (IRB approval No. SMC-2017-03-103-012), and written informed consent was obtained from all participants.

A total of 67 outpatients participated in the cognitive interviews (61 participants in Round 1 and six participants in Round 2). The mean age was 57.1 years (age range: 18 to 86 years, standard deviation: 15.7), 55.2% were men, 71.6% were married, and 50.7% had an educational level of below high school (Table 1). Of the study participants, 38.8% either had an educational level of below middle school or were aged above 65 years.

Although participants understood most of the K-PROMIS items during the cognitive interview in Round 1 (n = 61), there were several items that were difficult to comprehend. Fifty-four (20.2%) of the 268 K-PROMIS items were identified as difficult to comprehend for at least 20% of the respondents (Table 2). Among these 54 items, four items had both linguistic and cultural difficulties. Most K-PROMIS items involved linguistic difficulties, but items under “physical function” and “ability to participate in social roles and activities” involved difficulties with regard to both. Among the items that were difficult to comprehend, the most frequently identified K-PROMIS domain was “physical function” (24.5%). In particular, items which more than 60% of participants had difficulty in comprehending fell under “physical function” (10 items), “fatigue” (five items), and “ability to participate in social roles and activities” (one item).

Linguistic and cultural difficulties were specifically categorized into four (Table 3) and three (Table 4) issues, respectively. Most items with linguistic difficulties were related to “fatigue” (11 items) and “physical function” (11 items). In terms of the issue “Do not know the meaning of the word,” participants were unfamiliar with the foreign words or Chinese characters, such as “mental energy,” “restrict,” or “feel limited.” Regarding “Ambiguous meaning,” four items (“household chores,” “feel less alert,” “flights of stairs,” and “labor”) in the “fatigue” and “physical function” domains presented difficulties for 60% or more of the participants. For instance, for the item “How often were you too tired to do your household chores?” five out of six participants (83.3%) said that they understood the meaning of the term but had no experience with household chores. Most participants (85.7%) understood “flights of stairs” to mean “steps.” We conducted a retest by modifying the phrase to “climb five floors with several stairs,” but most participants still did not understand. Through further review, we modified the phrase “climb five floors with several stairs” to “climb up five floors.”

For the issues grouped under “Confused context,” the K-PROMIS domain that involved the most difficulties in comprehension was “physical function” (seven items). Most participants (85.7%) understood the items “walk up and down two steps” and “climb up five steps” to mean going up and down “two steps” and “five steps” at a time. We added the word “one step at a time” to make the intended meaning clearer, and then conducted the cognitive interview in Round 2. At this point, more than 80% of the participants understood the intended meaning.

Issues classified as “Others” were identified only under “fatigue.” We tried to use the same phrases or words that were previously translated in other domains in order to consistently match the meaning in whole domain in Korean. For instance, “How fatigued were you on average?” and “I feel fatigued” were translated into the Korean “How exhausted were you on average?” and “I was exhausted” in the FACIT-Fatigue scale. Of the participants, 66.7% said that there was no connection between “How fatigued were you” and “at its worst.”

Cultural difficulties were only identified in the “physical function” (seven items) and “ability to participate in social roles and activities” (one item) domains. In terms of the issue of “Differences in concept or unfamiliar activities,” regarding the item “I have to limit social activities at home,” five out of seven participants (71.4%) said that they do not socialize at home. This was because in Korea, “social activities” refer to activities involving interaction with people outside the home.

For the issue of “Differences in lifestyle and living environment,” all problematic items fell under “physical function.” As an item relevant to a different lifestyle, 85.7% of the participants did not understand what “cut food using eating utensils” meant. Americans cut food using eating utensils such as knives and forks, whereas Koreans use spoons and chopsticks. After discussing this issue with the PROMIS Statistical Center, we rephrased this item to “cut food using spoons or chopsticks.”

Regarding the issue of “Different metric units,” many participants (85.7%) indicated that they found it difficult to judge exactly what distance “two miles (3 km)” or “10 miles (16 km)” represented and that they needed suitable examples. Koreans are familiar with the concept of judging distances in terms of the number of bus stops or blocks on the way to one's destination, but the distances between bus stops or blocks can vary widely depending on the area of residence. For a similar reason, about 71.4% of the participants needed practical examples for measurement units like a “2 kg object” or “heavy object (5 kg).”

Of the 54 items, 25 were slightly revised after consultation with the PROMIS Statistical Center while 29 remained unchanged. These revised items were tested with an additional six participants in Round 2. During Round 2, most participants had no problems understanding the modified items. Three raters determined the types of issues associated with comprehension difficulties. Differences between raters were resolved by discussion until a consensus was reached.

In this study, we translated and linguistically validated the six adult profile domains of the PROMIS into Korean based on the standard FACIT multilingual translation methodology.27, 28 The majority of items in the K-PROMIS were well comprehended by Korean speakers, including older adults and those with lower educational levels. After two rounds of cognitive debriefing, we concluded that the K-PROMIS possesses satisfactory psychometric properties for use in clinical research and the care of Korean-speaking patients. However, despite adhering to rigorous procedures to maintain semantic and conceptual equivalence with the original PROMIS items, some linguistic and cultural issues persisted.

Overall, the prevalence of linguistic issues in all domains, with the exception of “pain intensity,” ranged from 10% to 20%. However, only 15 items (14.1%) under “physical function” and one item (3.2%) under “ability to participate in social roles and activities” had cultural issues. This might be because, by conferring with the PROMIS Statistical Center, we were able to eliminate common cultural issues identified in PROMIS linguistic validation studies from other countries during the translation process. For example, measurement units (e.g., miles, blocks), definitions of “leisure activities” and “recreational activities,” and extended definitions of “yard work” for people living in apartments were discussed during translation. Yet, cultural adaptations were particularly required for “physical function” items in different living environments (e.g., yard work, carrying a laundry basket up) and tools used in daily life (e.g., utensils, can openers). In terms of metrics, while we employed those used in Korea (e.g., km, kg), patients still experienced difficulties because they do no use metrics in daily life. In Korea, people are more likely use examples instead of exact metrics or units to express an amount or distance. For example, people prefer to use the number of steps or time taken to get to a destination (e.g., 30 minutes) than exact units (e.g., 1.5 km). This is similar to previous studies. In an Arabic version, “walking more than a mile” was replaced by “walking a distance more than a bus stop” because the mile is not a well-known measure in Arabic culture.34

The PROMIS was developed quantitatively with scientific accuracy for the purpose of making an evidence-based objective outcome measure available in clinical practice.35, 36 Yet, in this study, we found that individuals' experiences or situations affected their interpretations of the items. For example, regarding one item in the “fatigue” domain, “How often were you too tired to do your household chores?” some patients, especially older males, did not respond because they did not have experience of doing household chores. This is similar to a previous study, where patients could not answer some questions as they could not visualize the tasks/demands proposed.18 Thus, our study team considered including examples of household chores. However, the cognitive debriefing in the Dutch PROMIS validation study found that giving examples might lead to confusion or misunderstanding of items. For example, in the context of a question about physical activities, “Does your health now limit you in participating in active sports such as swimming, tennis, or basketball?,” patients were not able to think beyond these specific sports.37 Thus, after conferring with the PROMIS Statistical Center, we decided not to include examples or reference points. The issue of how patients' interpretations can vary based on such uncertainties should be further explored.

The PROMIS item bank covers a broad range of levels because the items need to be evaluated based on IRT methods.38 However, participants had difficulties distinguishing between similar idiomatic expressions and sentence constructions in the “fatigue” domain. There were 10 words (tired, fatigued, sluggish, run-down, physically drained, exhausted, bushed, totally drained, wiped out, extremely exhausted) but patients could not differentiate between them. This was even more problematic with people with limited vocabulary.23, 30 Translation of these 10 different words to describe increasing levels of fatigue may have resulted in a different ordering (item difficulties) of the items in the translated language, potentially introducing differential item functioning. Further language-specific item calibration would be necessary if important differential item functioning is found.

Similarly, there were some words/expressions that, though slightly different, were considered the same by participants. The study team acknowledged this issue during forward translation and tried to choose the translation with the greatest conceptual equivalence. However, participants still had difficulties in distinguishing between words. For example, participants could not differentiate “flight of stairs” from “steps,” “labor” from “work,” and “bending” from “stooping.” While not being able to differentiate between these words would not pose a problem in daily life, this ambiguity is detrimental to the evaluation based on IRT methods. It might be helpful for patients if they are provided with the exact definitions of words, or given pictures as guidance for words with ambiguous meanings.

Furthermore, participants, especially those with low educational levels, had difficulties with loanwords such as “energy” or “recreational activities,” but we had to use them because they had no Korean equivalents. These items could not be substituted because the suggestions offered were mostly colloquial. This issue has been mentioned in previous studies involving testing PRO measures such as the Common Terminology Criteria for Adverse Events that have been translated into Korean.39

This study has some limitations. First, as the setting was a single hospital, the participants were not representative of all Koreans. However, the sample possessed the demographic and clinical diversity to provide rigorous evidence of content validity in clinical settings. Therefore, additional testing to confirm its comprehensibility and cultural acceptability with regard to the general population with no experience of symptoms could be considered. Second, some items could not be modified because of the absence of Korean substitutes. Especially, four items included in the standard short forms were identified as difficult to comprehend for at least 20% of the respondents but items could not be modified as there were no suitable alternatives. Therefore, it is necessary to encourage clinicians and researchers who use the translated item banks in the target countries to share any culturally important content that might be missing. Furthermore, it might be reasonable to select different items in the standard short forms for use in Korea as these items could be problematic for Korean population. It is, therefore, necessary to encourage clinicians and researchers who use the translated item banks in the target countries to share any culturally important content that might be missing. Third, we conducted cognitive interviews with only six to seven participants per item; it might be better to increase this number to assess the level of comprehensibility and cognitive equivalence of the translation. However, this is higher than the number of participants recommended in the FACIT translation methodology for the debriefing of new items in the target country (five participants). In addition, we had an expert group who participated in the entire process, from linguistic validation to quality assurance to validation.

Regardless of these limitations, this study has several strengths. First, we performed purposive sampling to include people with low health literacy and cognitive abilities. Of the sample, 38.8% either had an educational level of high school or below or were older than 65 years, and we sought an approximately equal gender representation. Second, we included patients who were experiencing symptoms and difficulties included in the PROMIS items. This increased the validity of the study as individuals without symptoms might have had more difficulties in comprehending PRO items. For example, orthopedic patients would report problems with physical function more accurately than people who do not experience those limitations or symptoms. Next, we matched the words and phrases to increase the consistency of meaning within and between domains, except for the mental health profile, for which linguistic validation had already been performed by another Korean team.14 In addition, we designed common questionnaires and principles for the cognitive interviews and coding methods to reduce bias among interviewers.39 Lastly, we followed standard methodology and guidelines, including the item definitions provided by the PROMIS team, and tried to achieve consensus with the PROMIS Statistical Center throughout the process to ensure measurement invariance across languages.

Our study can contribute to the formation of a basis for the international measurement of PROs and formal process of PROMIS adoption across countries. Further psychometric validation of the K-PROMIS items will provide additional information regarding meaningful outcomes for chronic diseases and clinical settings.

Translated Korean PROMIS item banks for adults